Display responsive communication system and method

ABSTRACT

A multimedia communication system and computer-implemented method for transmitting auxiliary display content to an end-user communication device to be rendered on a display device with a special effect to emphasize an image included in the auxiliary display content, comprising a processor and a transmitter. The processor can be arranged to analyze image data included in an auxiliary display content, detect an object image or a background image in the auxiliary display content based on the analysis of the image data, determine a special effect based on the analysis of the image data, and apply the special effect to the auxiliary display content to modify display properties for the auxiliary display content such that the object image is emphasized or pops-out. The transmitter can be arranged to send the auxiliary display content with modified display properties to an end-user communication device. The special effect can comprise a non-customization special effect, a simple foreground special effect or a selective foreground special effect.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part and claims priority to U.S.patent application Ser. No. 29/745,050, filed on Aug. 3, 2020, which ishereby incorporated by reference as if set forth in its entirety herein.

FIELD OF THE DISCLOSURE

The present disclosure relates to a communication system and method,and, more particularly, to a communication system andcomputer-implemented method that can analyze display content andselectively modify display properties of the content to enhance displayeffects.

BACKGROUND OF THE DISCLOSURE

In a computer networked environment such as the Internet, contentproviders typically supply content for display on end-user communicationdevices. The content generally includes images, text, video or audio.The content can be linked to webpages associated with the contentproviders. Frequently, the content includes image data that, whendisplayed on the end-user communication devices, lacks desired effectssuch as, for example, adequate depth perception or focus on a particularsubject in the content. There exists a need for a technological solutionthat can analyze display content and selectively modify displayproperties to enhance one or more display effects.

SUMMARY OF THE DISCLOSURE

In some embodiments, aspects of the disclosure provide a technologicalsolution that can analyze multimedia content and selectively modifydisplay properties based on auxiliary display (AD) content to enhanceone or more display effects. Some aspects of the disclosure includes asystem and a computer-implemented method for analyzing the AD contentand selectively modifying display properties to enhance display effectsuch as, for example, depth perception of an image, highlighting of amain subject of the image, and bringing together various portions of theAD content to display a unified and wholistic rendering of the ADcontent.

According to a nonlimiting embodiment, the system comprises a multimediacommunication system for transmitting auxiliary display content to anend-user communication device to be rendered on a display device with aspecial effect to emphasize an image included in the auxiliary displaycontent. The multimedia communication system comprises a processor thatis arranged to: analyze image data included in an auxiliary displaycontent; detect an object image or a background image in the auxiliarydisplay content based on the analysis of the image data; determine aspecial effect based on the analysis of the image data; and apply thespecial effect to the auxiliary display content to modify displayproperties for the auxiliary display content such that the object imageis emphasized or pops-out. The system can further comprise a transmitterarranged to send the auxiliary display content with modified displayproperties to an end-user communication device. The special effect cancomprise a non-customization special effect, a simple foreground specialeffect or a selective foreground special effect.

In the multimedia communication system, the processor can be arrangedto: generate or modify a multimedia content rendering application; andsend the multimedia content rendering application to the end-usercommunication device, wherein the multimedia content renderingapplication is arranged to, when executed by a processor in the end-usercommunication device, render the auxiliary display content with modifieddisplay properties on a display device.

In the multimedia communication system, the non-customization specialeffect can comprise an overlay border that, when rendered by theend-user communication device, is displayed around an inner-border of anauxiliary display content display window on a display device.

In the multimedia communication system, the overlay border can have atleast one corner and a non-overlay portion that emphasizes the objectimage in the auxiliary display content.

In the multimedia communication system, the simple foreground specialeffect can comprise an overlay border having an overlay portion and anon-overlay portion that, when rendered by the end-user communicationdevice, is displayed as an overlay border atop of the background imagein an auxiliary display content display window, with the object imagebeing displayed without any overlay in the auxiliary display contentdisplay window.

In the multimedia communication system, the overlay border can have atleast one corner and the non-overlay portion is rendered to emphasizethe object image or an action depicted in the auxiliary display content.

In the multimedia communication system, the selective foreground specialeffect can comprise an overlay border having an overlay portion and anon-overlay portion that, when rendered by the end-user communicationdevice, is displayed as an overlay border atop of the background imagein an auxiliary display content display window, with at least a portionof the object image being displayed without any overlay in the auxiliarydisplay content display window.

In the multimedia communication system, the selective foreground specialeffect can comprise a zoom-in or a zoom-out of the object image, or ablurring of the background image when rendered by the end-usercommunication device.

In the multimedia communication system, the selective foreground specialeffect can comprise a blurring of the background image.

In the multimedia communication system, the selective foreground specialeffect can comprise zooming-in or zooming-out the object image whenrendered by the end-user communication device.

According to a further limiting embodiment, a computer-implementedmethod is provided for transmitting auxiliary display content to anend-user communication device to be rendered on a display device with aspecial effect to emphasize an image included in the auxiliary displaycontent. The method can comprise: analyzing, by a processor, image dataincluded in an auxiliary display content; detecting, by the processor,an object image and a background image in the auxiliary display content;determining, by the processor, a special effect based on the analysis ofthe image data; and applying, by the processor, the special effect tothe auxiliary display content to modify display properties for theauxiliary display content to emphasize the object image, wherein thespecial effect comprises a non-customization special effect, a simpleforeground special effect or a selective foreground special effect.

The method can further comprise: creating a multimedia content renderingapplication; and sending the multimedia content rendering application toan end-user communication device, wherein the multimedia contentrendering application is arranged to, when executed by a processor inthe end-user communication device, render the auxiliary display contentwith modified display properties on a display device.

In the method, the non-customization special effect can comprise anoverlay border that, when rendered by an end-user communication device,is displayed around an inner-border of an auxiliary display contentdisplay window on a display device.

In the method, the overlay border can have at least one corner and anon-overlay portion that emphasizes the object image in the auxiliarydisplay content.

In the method, the simple foreground special effect can comprise anoverlay border having an overlay portion and a non-overlay portion that,when rendered by an end-user communication device, is displayed as anoverlay border atop of the background image in an auxiliary displaycontent display window, with the object image being displayed withoutany overlay in the auxiliary display content display window.

In the method, the overlay border can have at least one corner and thenon-overlay portion is rendered to emphasize the object image or anaction depicted in the auxiliary display content.

In the method, the selective foreground special effect can comprise anoverlay border having an overlay portion and a non-overlay portion that,when rendered by an end-user communication device, is displayed as anoverlay border atop of the background image in an auxiliary displaycontent display window, with at least a portion of the object imagebeing displayed without any overlay in the auxiliary display contentdisplay window.

In the method, the selective foreground special effect can comprise azoom-in or a zoom-out of the object image when rendered by an end-usercommunication device.

In the method, the selective foreground special effect can comprise ablurring of the background image when rendered by the end-usercommunication device.

In the method, the selective foreground special effect can comprisezooming-in or zooming-out the object image when rendered by the end-usercommunication device.

Additional features, advantages, and embodiments of the disclosure maybe set forth or apparent from consideration of the detailed descriptionand drawings. Moreover, it is to be understood that the foregoingsummary of the disclosure and the following detailed description anddrawings provide nonlimiting examples that are intended to providefurther explanation without limiting the scope of the disclosure asclaimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosure, are incorporated in and constitute apart of this specification, illustrate embodiments of the disclosure andtogether with the detailed description serve to explain the principlesof the disclosure.

FIG. 1 is a block diagram depicting an implementation of a multimediacommunication system in a user environment, arranged according to theprinciples of the disclosure.

FIG. 2 shows a nonlimiting example of multimedia content that can bereceived by an end-user communication device and displayed on a displayscreen of the device.

FIG. 3 is a block diagram depicting a nonlimiting embodiment of acontent provider (CP) communication device.

FIG. 4 is a block diagram depicting a nonlimiting embodiment of theend-user communication device.

FIG. 5 is a block diagram depicting a nonlimiting embodiment of amultimedia content modifier (MCM) system.

FIG. 6 is a diagram depicting a nonlimiting example of an operation ofan overlay processor included in the MCM system, shown in FIG. 5.

FIG. 7 is a diagram depicting a nonlimiting example of a modification ofauxiliary display (AD) content performed by the overlay processor.

FIG. 8 is a diagram depicting a nonlimiting embodiment of a process foranalyzing AD content and selectively modifying display properties basedon image data in the AD content.

FIGS. 9-15 and 16A-16C are diagrams depicting additional examples ofmodifications of AD content performed by the overlay processor.

The present disclosure is further described in the detailed descriptionthat follows.

DETAILED DESCRIPTION OF THE DISCLOSURE

The disclosure and its various features and advantageous details areexplained more fully with reference to the nonlimiting embodiments andexamples that are described or illustrated in the accompanying drawingsand detailed in the following description. It should be noted thatfeatures illustrated in the drawings are not necessarily drawn to scale,and features of one embodiment can be employed with other embodiments asthose skilled in the art would recognize, even if not explicitly stated.Descriptions of well-known components and processing techniques may beomitted so as to not unnecessarily obscure the embodiments of thedisclosure. The examples are intended merely to facilitate anunderstanding of ways in which the disclosure can be practiced and tofurther enable those skilled in the art to practice the embodiments ofthe disclosure. Accordingly, the examples and embodiments should not beconstrued as limiting the scope of the disclosure. Moreover, it is notedthat like reference numerals represent similar parts throughout theseveral views of the drawings.

FIG. 1 is a block diagram depicting an implementation of a multimediacommunication system 10 in an environment 1, arranged according to theprinciples of the disclosure. The environment 1 can include a network20, a content provider (CP) communication device 30 and one or moreend-user communication devices 40, any or all of which can be coupled toeach other directly or via the network 20 through one or morecommunication links 5.

The multimedia communication system 10 can include a computer, a server,a storage device, a database, an interface or a communication device.The multimedia communication system 10 can include a multimedia contentmodifier (MCM) system 100 (shown in FIG. 5). The multimediacommunication system 10 can include the communication devicearchitecture 300 (shown in FIG. 3) or 350 (shown in FIG. 4). Themultimedia communication system 10 can be configured to providemultimedia content to the end-user communication devices 40. Themultimedia content can include main display (MD) content and auxiliarydisplay (AD) content that can be rendered on the end-user communicationdevice 40. The AD content can be created, modified, or provided by theCP communication device 30. The AD content can be received by theend-user communication device 40 from the multimedia communicationsystem 10 or CP communication device 30.

FIG. 2 shows a nonlimiting example of the multimedia content that can bereceived by the end-user communication device 40 and displayed on adisplay screen 50 of a display device (for example, display device 354,shown in FIG. 4). In this nonlimiting example, the multimedia contentincludes main display (MD) content 51 and auxiliary display (AD) content52. The multimedia content (including MD content and AD content) can bereproduced by the end-user communication device 40, for example, via amultimedia content rendering (MCR) application executed by a processor(for example, processor 352, shown in FIG. 4) in the device 40. Themultimedia content can include any displayable or audible content thatcan be reproduced by the end-user communication device 40. The ADcontent 52 can be received without the MD content 51 and displayed onthe end-user communication device 40, according to the principles of thedisclosure.

The MCR application can be generated or modified by the multimediacommunication system 10 and downloaded to each end-user communicationdevice 40. The MCR application can include, for example, a browserapplication that can be installed and/or run on the end-usercommunication device 40. The MCR application can, when executed by theprocessor 352 (shown in FIG. 4), convert, for example, HyperText MarkupLanguage (HTML), Cascading Style Sheets and JavaScript into a workingwebsite or webpage that can be interacted with by an operator of theend-user communication device 40, such as, for example, through a userinterface.

The MD content 51 can include, for example, video/audio content,streaming video/audio content, or a webpage or website, or a resultspage or a landing page of a search operator.

The auxiliary display (AD) content 52 can include, for example, acontent provider source identifier, text, an image, a logo, a video, auniform resource locator (URL) address, or a sound. The AD content 52can include, for example, an advertisement, a pop-up or pop-outadvertisement, a message, a picture-in-picture image or message, abanner advertisement or any other content that can be included in,superimposed onto or reproduced (including displayed) with the MDcontent 51, or rendered or displayed by itself on the end-usercommunication device 40, without the MD content 51.

In the nonlimiting example shown in FIG. 2, the AD content 52 includes apop-up or pop-out advertisement that includes an image 32 of a buildingstructure and a textual message 34 relating to the image 32. The ADcontent 52 can include instructions and data that, when executed by theprocessor 352 (shown in FIG. 4), cause the end-user communication device40 to render the AD content 52 on the display screen 50 of the displaydevice 352 (shown in FIG. 4). The AD content 52 can include instructionsto display the AD content 52 in two or more display regions 53, 54. Thedisplay region 52 can include the image data in the AD content, such as,for example, the image 32 with background; and, the display region 54can include the non-image data in the AD content, such as for example,the textual message 34 and one or more call-to-action selectors 56. TheAD content 52 can include instructions and data to render a specialeffect 57, which can include, for example, a border effect. Forinstance, the instructions and data can be configured to cause theend-user communication device 40 to render and display the specialeffect 57 as a border effect along an inner border of the AD content 52(shown in FIG. 2), or another special effect such as, for example, ablurring effect, a rotating circle, a zoom-in effect, a zoom-out effect,a fade-in or a fade-out effect, or any other effect that mightemphasize, highlight or “bring-to-life” the main subject (for example,image 32, shown in FIG. 2) in the AD content.

As seen in FIG. 2, the AD content can include instructions and data torender one or more call-to-action selectors 56 on the display screen 50.In a nonlimiting embodiment, a portion of the image 32 or textualmessage 34 can include a call-to-action. The selector 56 can include aradio button, a hyperlink, or a selectable item that can trigger orinitiate a call-to-action by the end-user communication device 40, suchas, for example, to access a file or a particular website or webpage onthe Internet. The call-to-action selector 56 can be linked to a webpageor a website belonging to the content provider, which can be hosted, forexample, by the multimedia communication system 10 or the CPcommunication device 30 (shown in FIG. 1).

The AD content can include instructions (and data) that, when executedby the processor 352 (shown in FIG. 4), causes the end-usercommunication device 40 to render the special effect 57, including, forexample, an overlay having a predetermined shape, size, texture, coloror location, or another special effect such as, for example, theblurring effect, rotating circle, zoom-in effect, zoom-out effect,fade-in effect, or fade-out effect, among others, that can emphasize orhighlight the main subject (for example, image 32, shown in FIG. 2) inthe AD content.

In the nonlimiting example seen in FIG. 2, the special effect 57includes a gray-colored rectangular shape border overlay located withinthe perimeter of the displayed AD content 52. However, the specialeffect 57 can include a circle, oval, triangle or any other shape thatcan emphasize or “bring-to-life” the subject (for example, image 32,shown in FIG. 9). Moreover, the special effect 57 can be animated, suchas, for example, a spinning circle (not shown), or portions of the imagecontent (for example, image 32) can be zoomed-in, zoomed-out, blurred,brought into focus, vibrated, or otherwise manipulated to grab theuser's attention when viewing the display on the end-user communicationdevice 40. The layout of the display regions 53, 54 or special effect 57can be arranged to emphasize the image 32, textual message 34 andcall-to-action 56, as these are likely to be “clicked” by users.

Referring to FIG. 1, the multimedia communication system 10 (or CPcommunication device 30) can include a search engine server of a searchengine operator that operates a search engine website. The MD content 51(shown in FIG. 2) of a search engine webpage, such as, for example, asearch results webpage or a landing webpage, can include the results ofa search or the landing webpage displayed with the AD content 52included, such as, for example, in a content slot on the search resultswebpage or landing webpage.

The CP communication device 30 can include the communication devicearchitecture 300, shown in FIG. 3. The CP communication device 30 caninclude the MCM system 100, shown in FIG. 5. The CP communication device30 can be operated, for example, by a content provider, to create,modify or provide AD content to be reproduced by the end-usercommunication devices 40.

FIGS. 3 and 4 show nonlimiting examples of a communication devicearchitecture 300 and 350, respectively, including associated computingresource assets that can be used to implement the systems and methodsdescribed herein. The communication device architecture 300 (or 350) canbe included in the CP communication device 30 (shown in FIG. 1); and,the communication device architecture 350 (or 300) can be included inthe end-user communication device 40 (shown in FIG. 1). The computingresource assets shown here, their connections and relationships, andtheir functions, are meant to be exemplary only, and are not meant tolimit implementations of the disclosure.

The communication device architecture 300 includes a processor 302,memory 304, a storage device 306, a high-speed interface 308 connectingto memory 304 and high-speed expansion ports 310, and a low speedinterface 312 connecting to low speed bus 314 and storage device 306.Each of the components 302, 304, 306, 308, 310, and 312, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 302 canprocess instructions for execution within the CP communication device30, including instructions stored in the memory 304 or on the storagedevice 306 to display graphical information for a graphic user interface(GUI) on an external input/output device, such as display 316 coupled tohigh speed interface 308. In other implementations, multiple processorsand/or multiple buses can be used, as appropriate, along with multiplememories and types of memory. Also, CP communication devices 30 can beconnected, with each device providing portions of the necessaryoperations (for example, as a server bank, a group of blade servers, ora multi-processor system).

The memory 304 stores information within the CP communication device 30.In one implementation, the memory 304 is a volatile memory unit orunits. In another implementation, the memory 304 is a non-volatilememory unit or units. The memory 304 can also be another form ofcomputer-readable medium, such as a magnetic or optical disk.

The storage device 306 is capable of providing mass storage for the CPcommunication device 30. In one implementation, the storage device 306can be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid-state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied in anon-transitory computer-readable medium. The computer program productcan contain instructions that, when executed, perform one or moremethods, such as those described above. The computer-readable medium caninclude an information carrier such as the memory 304, the storagedevice 306, or memory on processor 302.

The high-speed controller 308 manages bandwidth-intensive operations forthe CP communication device 30, while the low speed controller 312manages lower bandwidth-intensive operations. Such allocation offunctions is exemplary only. In one implementation, the high-speedcontroller 308 is coupled to memory 304, display 316 (for example,through a graphics processor or accelerator), and to high-speedexpansion ports 310, which may accept various expansion cards (notshown). In the implementation, low-speed controller 312 is coupled tostorage device 306 and low-speed expansion port 314. The low-speedexpansion port, which may include various communication ports (forexample, USB) may be coupled to one or more input/output devices, suchas a keyboard, a pointing device, a scanner, or a networking device suchas a switch or router, for example, through a network adapter.

The CP communication device 30 can be implemented in a number ofdifferent forms, as seen in FIG. 3. For example, it can be implementedas a standard server 320, or multiple times in a group of such servers.It can also be implemented as part of a rack server system 324. Inaddition, it can be implemented in a personal computer such as a laptopcomputer 322. Alternatively, computing resource assets from the CPcommunication device 30 can be combined with other computing resourceassets in a mobile device, such as the end-user communication device 40.Each of such devices can contain one or more of the devices 30, 40, andan entire system may be made up of multiple devices 30, 40 communicatingwith each other through communication links.

As seen in FIG. 4, the communication device architecture 350 includes aprocessor 352, memory 364, an input/output device such as a display 354,a communication interface 366, and a transceiver 368, among othercomputing resource assets. The device architecture 350 can also beprovided with a storage device, such as a microdrive or other device, toprovide additional storage. Each of the computer resource assets 352,364, 354, 366, and 368, are interconnected using various communicationlinks, buses, and several of the computing resource assets can bemounted on a common motherboard or in other manners as appropriate.

The processor 352 can execute instructions within the end-usercommunication device 40, including instructions stored in the memory364. The processor can be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor canprovide, for example, for coordination of the other computing resourceassets of the device 40, such as control of user interfaces,applications run by device 40, and wireless communication by device 40.

Processor 352 can communicate with a user through control interface 358and display interface 356 coupled to a display 354. The display 354 canbe, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display)or an OLED (Organic Light Emitting Diode) display, or other appropriatedisplay technology. The display interface 356 can comprise appropriatecircuitry for driving the display 354 to present graphical and otherinformation to a user. The control interface 358 can receive commandsfrom a user and convert them for submission to the processor 352. Inaddition, an external interface 362 can be provided in communicationwith processor 352, so as to enable near area communication of device 40with other devices. External interface 362 can provide, for example, forwired communication in some implementations, or for wirelesscommunication in other implementations, and multiple interfaces can alsobe used.

The memory 364 stores information within the end-user communicationdevice 40. The memory 364 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. Expansion memory 374 can also beprovided and connected to device 40 through expansion interface 372,which can include, for example, a SIMM (Single In Line Memory Module)card interface. Such expansion memory 374 can provide extra storagespace for device 40, or can also store applications or other informationfor device 40. Specifically, expansion memory 374 can includeinstructions to carry out or supplement the processes described above,and can include secure information also. Thus, for example, expansionmemory 374 can be provided as a security module for device 40, and canbe programmed with instructions that permit secure use of device 40. Inaddition, secure applications can be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 364, expansionmemory 374, or memory on processor 352, which can be received, forexample, over transceiver 368 or external interface 362.

The end-user communication device 40 can communicate wirelessly throughcommunication interface 366, which can include digital signal processingcircuitry where necessary. Communication interface 366 can provide forcommunications under various modes or protocols, such as GSM voicecalls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, orGPRS, among others. Such communication can occur, for example, throughradio-frequency transceiver 368. In addition, short-range communicationcan occur, such as using a BLUETOOTH, WiFi, or other such transceiver(not shown). In addition, GPS (Global Positioning System) receivermodule 370 can provide additional navigation- and location-relatedwireless data to device 40, which can be used as appropriate byapplications running on device 40.

The end-user communication device 40 can also communicate audibly usingaudio codec 360, which can receive spoken information from a user andconvert it to usable digital information. Audio codec 360 can likewisegenerate audible sound for a user, such as through a speaker, forexample, in a handset of device 40. Such sound can include sound fromvoice telephone calls, can include recorded sound (for example, voicemessages, music files, etc.) and can also include sound generated byapplications operating on device 40.

The communication device architecture 300 can be implemented in a numberof different forms, as shown in FIG. 3. For example, it can beimplemented as a cellular telephone 380. It can also be implemented aspart of a smart phone 382, personal digital assistant, or other similarmobile device.

Referring to FIGS. 1 and 4, the end-user communication device 40 can bearranged to communicate via the network 20 and to display data such asthe multimedia content on the display screen 50 of the display device354 (shown in FIG. 4). Using the user interface on the end-usercommunication device 40, a user can submit requests to receivemultimedia content. The requests can be requests to a search engine andthe requests can include search queries. In some implementations, therequests can include a request to access a webpage or website. Themultimedia content can be received by the end-user communication device40 from the multimedia communication system 10. The received multimediacontent can include AD content, which may have originated from the CPcommunication device 30.

The end-user communication device 40 can reproduce the multimediacontent on the display device 352 (shown in FIG. 4), including any ADcontent that might have been included in the multimedia content. Forinstance, the end-user communication device 40 can display the MDcontent 51 (shown in FIG. 2) or a results webpage or a landing webpage(not shown), filling one or more content slots in or on the MD contentwith the AD content 52 (shown in FIG. 2). The AD content 52 can bedisplayed by itself, without the MD content 51. The content slots can bedynamic and can be automatically adjusted in terms of position, layout,shape, or size on the display screen 50 (shown in FIG. 2). The MCRapplication can be stored in the multimedia communication system 10, inthe end-user communication device 40, or elsewhere in the environment 1.

The MCR application can include markup language annotations foridentifying content and creating structured documents, including images,text, links, sounds, and other objects. The markup language annotationscan include a plurality of tags for displaying multimedia content,including, for example, the MD content 51 and/or AD content 52 (shown inFIG. 2) on the display screen 50 of the end-user communication device40. The markup language can include, for example, Standard GeneralizedMarkup Language (SGML), Scalable Vector Graphics (SVG), HTML, ExtensibleMarkup Language (XHTML or XML), XML User Interface Language (XUL), orLaTeX. The markup language annotations can be provided as a markuplanguage file that can be executed by, for example, a web browserrunning in the end-user communication device 40 to render the multimediacontent on the display device 354 (shown in FIG. 4). The MCR applicationcan include the web browser. Alternatively, the web browser can includethe MCR application.

The MCR application can include style sheet language annotations forproviding rules for stylistics and for describing the presentation ofthe content and document with the markup language annotations, such as,for example, the markup language file. The style sheet languageannotations can include, for example, colors, fonts, layouts, and otherstylistic properties. The style sheet language can include, for example,CSS, Document Style Semantics and Specification Language (DSSSL), orExtensible Stylesheet Language (XSL). The style sheet languageannotations can be provided as a style sheet language file.Alternatively, the style sheet language annotations can be incorporatedinto the file containing the markup language annotations.

The MCR application can include scripting language instructions tocreate interactive effects related to the markup language annotations orstyle sheet language annotations. The scripting language can include,for example, Bash (for example, for Unix operating systems), ECMAScript(or JavaScript) (for example, for web browsers), Visual Basic (forexample, for Microsoft applications), Lua, or Python. The scriptinglanguage instructions can include instructions that when executed by,for example, the web browser on the end-user communication device 40effect rendering of multimedia content, as seen in the Figures,including, for example, FIG. 2, 7 or 9-15, on the display device 354(shown in FIG. 4). The scripting language instructions can be providedas a scripting language file. Alternatively, the scripting languageinstructions can be incorporated into the file containing the markuplanguage annotations.

The MCR application can include a document object model (DOM) such asfor HTML or XML (for example, DOM5 HTML) that can create object-orientedrepresentations of the content or documents that can be modified withthe scripting language instructions. A DOM includes a cross-platform andlanguage-independent convention for representing and interacting withobjects in HTML, XHTML/XML, SGML, SVG, or XUL. As used herein, adocument can refer to the DOM's underlying document.

The MCR application can be configured to be executable by the processor352 (shown in FIG. 4) and can follow a model-view-controller (MVC)design pattern for user interfaces. According to the MVC design pattern,an application can be divided into three areas of responsibility,including: (1) the Model, which includes the domain objects or datastructures that represent the application's state; (2) the View, whichobserves the state and generates an output to the users; and, (3) theController, which translates user input into operations on the model.

FIG. 5 is a block diagram depicting a nonlimiting embodiment of themultimedia content modifier (MCM) system 100, constructed according tothe principles of the disclosure. The MCM system 100 can be configuredto implement the various aspects of the embodiments described. The MCMsystem 100 can be included in any one or more of the multimediacommunication system 10, CP communication device 30 or end-usercommunication devices 40. The MCM system 100 can include (or can beincluded in) the communication device architecture 300 (shown in FIG. 3)or 350 (shown in FIG. 4). The MCM system 100 can include a suite ofcomputer resources 130 to 180 that can be download to and/or installedon the multimedia communication system 10, or the end-user communicationdevices 40.

A nonlimiting embodiment of the disclosure can include the MCM system100, which can be arranged to create or modify AD content to enhance thedepth perception of an image, highlight the main subject of the imageand to bring together the lower part of an AD content format (with thetext and call to action) and an upper part with brand or other sourceidentifier information and the image. Through creation or modificationof the AD content, image rendering can be provided that improvesvisibility and/or recognizability of AD content on the end-usercommunication device 40, including making it easier for a user toidentify and effectively select a call-to action selector 56. Thedisplay of images can be enhanced in or by the AD content to make anaction or an object “pop-out” to the user.

For instance, a special effect can be implemented such as a smart borderin the AD content via the MCM system 100, such that, for example: (1) itdoesn't modify a border if there isn't enough border to show; (2)doesn't overlay human faces; (3) doesn't overlay featured products; (4)enhances depth by selectively overlaying the border on part of thescene, subject or product; (5) enhances the scene action or movement bytaking into consideration the orientation of the action displayed (forexample, car going from right to left, as seen in FIG. 9). The MCMsystem can be arranged to implement other special effects such aszoom-in/out, blurring, or other image manipulation to emphasize the mainsubject, for example, as seen in FIG. 11.

The MCM system 100 includes an overlay processor 160, which can includea machine learning (ML) platform. The MCM system can include a processor110, a storage 120, a network interface 130, an input-output (10)interface 140, a driver suite 150, or a multimedia content renderer 170.The MCM system 100 can include an overlay model training and tuning(OMTT) unit 180. The overlay processor 160 and OMTT unit 180 can becomprised in a single computing resource asset, as seen in FIG. 5, orprovided as separate computing resource assets (not shown).

The MCM system 100 can include a bus 105, which can be connected to anyor all of the computer resource assets 110 to 180 by communicationlinks. The bus 105 can be connected to the architectures 300 or 400(shown in FIGS. 3 and 4).

Any one or more of the computing resource assets 110 to 180 can includea unique and separate computing device, as seen in FIG. 5, or two ormore of the computing resource assets can be integrated or integratableas a single computing device. Any of the computing resource assets 120to 180 can include a computer resource that can be executed on theprocessor 110 as one or more computing resource processes. The computerresources can be contained in the storage 120.

The bus 105 can include any of several types of bus structures that canfurther interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures.

The processor 110 can include any of various commercially availableprocessors. The processor 110 can include a computing device. Dualmicroprocessors and other multi-processor architectures can be employedas the processor 110. The processor 110 can include a central processingunit (CPU) or a graphic processing unit (GPU). The processor 110 can bearranged to interact with any of the computer resource assets 120 to 180to carry out or facilitate with the processes described herein. Theprocessor 110 can be further arranged to interact with the computerresource assets in the architectures 300 and 400 (shown in FIGS. 3 and4).

The storage 120 can include a read-only memory (ROM) 120A, arandom-access memory (RAM) 120B, a hard disk drive (HDD) 120C, anoptical disk drive (ODD) 120D, and a database (DB) 120E. The storage 120can provide nonvolatile storage of data, data structures, andcomputer-executable instructions, and can accommodate the storage of anydata in a suitable digital format.

The storage 120 can include a computer-readable medium that can holdexecutable or interpretable computer code (or instructions) that, whenexecuted by the processor 110, cause the steps, processes and methods inthis disclosure to be carried out. The computer-readable medium can becontained in the HDD 120C or ODD 120D. The computer readable medium caninclude sections of computer code that, when executed by the processor110, cause the MCM system 100 to analyze display content, identifyobjects (for example, people, animals, or products) and things (forexample, background, or scenery), and create or apply a special effect(for example, special effect 57, shown in FIG. 9) based on the displaycontent, for example, such that an object is perceived by a user of theend-user communication device 40 or creates a perception that the objectis popping out of the display content or coming to life. The MCM system100 can modify display properties for the display content such that, forexample, the image of the object zooms in or out, so as to emphasize theobject or make it a focal point in the display content.

A basic input-output system (BIOS) can be stored in the non-volatilememory 120A, which can include, for example, a ROM, an erasableprogrammable read-only memory (EPROM), or an electrically erasableprogrammable read-only memory (EEPROM). The BIOS can contain the basicroutines that help to transfer information between any one or more ofthe computing resource assets 110 to 180 in the system 100, such asduring start-up.

The RAM 120B can include dynamic random-access memory (DRAM), asynchronous dynamic random-access memory (SDRAM), a static random-accessmemory (SRAM), or another high-speed RAM for caching data.

The HDD 120C can include, for example, an enhanced integrated driveelectronics (EIDE) drive, a serial advanced technology attachments(SATA) drive, or any suitable hard disk drive for use with big data. TheHDD 120C can be configured for external use in a suitable chassis (notshown).

The ODD 120D can be arranged to read or write from or to a compact disk(CD)-ROM disk (not shown), or, read from or write to other high capacityoptical media such as a digital versatile disk (DVD).

The HDD 120C or ODD 120D can be connected to the bus 105 by a hard diskdrive interface (not shown) and an optical drive interface (not shown),respectively. The hard disk drive interface (not shown) can include aUniversal Serial Bus (USB) (not shown), an IEEE 1394 interface (notshown), and the like, for external applications.

The DB 120E can store training datasets and testing data sets, forexample, for an overlay model used by the machine learning platform inthe overlay processor 160. The training datasets can be updatedperiodically (or continuously) with updated parametric values, such as,for example, during parametric tuning of the overly model.

Any number of computer resources can be stored in the storage 120,including, for example, a program module, an operating system, anapplication program, an application program interface (API), or programdata. The computing resource can include an API such as, for example, aweb API, a simple object access protocol (SOAP) API, a remote procedurecall (RPC) API, a representation state transfer (REST) API, or any otherutility or service API. Any (or all) of the operating system,application programs, APIs, program modules, and program data can becached in the RAM 120B as executable sections of computer code.

The API can include an API for a markup language such as, for example,SGML, SVG, HTML, XHTML/XML), XUL, or LaTeX.

The API can include an API for a style sheet language, such as, forexample, CSS, DSSSL, or XSL. The API can include a web-based API, anoperating system API, a database system API, a computer hardware API, ora library API. The API can include, for example, one or more of the APIsavailable at <<https://developers.google.com>>.

The API can include one or more APIs that connect webpages to scripts orprograming languages, including modelling (for example, SGML, SVG, HTML,XHTML/XML, XUL) documents as objects.

The API can include a document object model (DOM) API, such as for HTMLor XML (for example, DOM5 HTML), that can create object-orientedrepresentations of webpages that can be modified with a scripting modulein the multimedia content renderer 170 (or overlay processor 160). A DOMcan include a cross-platform and language-independent convention forrepresenting and interacting with objects in HTML, XHTML/XML, SGML, SVG,or XUL.

The network interface 130 can be connected to the network 20 (shown inFIG. 1), which can include the Internet. The network interface 130 caninclude a wired or a wireless communication network interface (notshown) or a modem (not shown). When used in a local area network (LAN),the MCM system 100 can be connected to the LAN network through the wiredor wireless communication network interface; and, when used in a widearea network (WAN), the MCM system 100 can be connected to the WANnetwork through the modem. The modem (not shown) can be internal orexternal and wired or wireless. The modem can be connected to the systembus 105 via, for example, a serial port interface (not shown). Thenetwork interface 130 can include a receiver (not shown), a transmitter(not shown) or a transceiver (not shown).

The input-output (10) interface 140 can receive commands or data from anoperator via a user interface (not shown), such as, for example, akeyboard (not shown), a mouse (not shown), a pointer (not shown), astylus (not shown), a microphone (not shown), a speaker (not shown), ora display device (not shown). The received commands and data can beforwarded from the IO interface 140 as instruction to data signals, viathe bus 105, to any of the computer resource assets 110 to 180.

The driver suite 150 can include an audio driver 150A and a video driver150B. The audio driver 150A can include a sound card, a sound driver(not shown), an interactive voice response (IVR) unit, or any otherdevice necessary to render a sound signal on a sound production device(not shown), such as for example, a speaker (not shown). The videodriver 150B can include a video card (not shown), a graphics driver (notshown), a video adaptor (not shown), or any other device necessary torender an image signal on a display device (not shown).

The overlay processor 160 can include a supervised machine learningplatform or an unsupervised machine learning platform. The machinelearning platform can include, for example, a Word2vec deep neuralnetwork, a convolutional architecture for fast feature embedding(CAFFE), an artificial immune system (AIS), an artificial neural network(ANN), a convolutional neural network (CNN), a deep convolutional neuralnetwork (DCNN), region-based convolutional neural network (R-CNN),you-only-look-once (YOLO), a Mask-RCNN, a deep convolutionalencoder-decoder (DCED), a recurrent neural network (RNN), a neuralTuring machine (NTM), a differential neural computer (DNC), a supportvector machine (SVM), a deep learning neural network (DLNN), NaiveBayes, decision trees, logistic model tree induction (LMT), NBTreeclassifier, case-based, linear regression, Q-learning, temporaldifference (TD), deep adversarial networks, fuzzy logic, K-nearestneighbor, clustering, random forest, rough set, or any other machineintelligence platform capable of supervised or unsupervised learning foranalyzing and identify or predicting objects and things in multimediacontent. The machine learning platform can include an overlay machinelearning (ML) model (or overlay model) that can analyze image data inauxiliary display (AD) content, identify objects and things, and createor apply special effects to the AD content based on the image data inthe AD content (for example, AD content 52, shown in FIG. 2).

The overlay processor 160 is arranged to receive AD content (forexample, AD content 52, shown in FIG. 2). The overlay processor 160 canbe arranged to parse the AD content from received multimedia content.The overlay processor 160 is arranged to analyze image data in the ADcontent, extract features, classify the extracted features, cluster orgroup results and identify (or predict) objects and background in theimage data. The overlay processor 160 can be arranged to detect andanalyze any existing special effects (for example, border, shown in FIG.2) included in the AD content, or to create and apply the special effectto the AD content, depending on the identified (or predicted) objects orbackground. The overlay processor 160 is further arranged to create ormodify the special effect and/or image data to emphasize, bring into theforeground or highlight the object (or background), for example, suchthat the object or foreground pops out or appears to come to life fromthe rest of the display content when rendered on the end-usercommunication device 40.

The overlay processor 160 can be arranged (for example, through trainingof the overlay model) to, among other things: not modify an existingspecial effect 57, such as, for example, when the special effect 57includes an overlay border and a determination is made that there isinsufficient overlay to be rendered on the display screen; not overlayobjects (discussed below) that are to be emphasized, brought into theforeground or highlighted; enhance depth by rendering the special effect57 on a part of the object or background (discussed below); and, enhancean action or movement in the AD content by analyzing and identifying (orpredicting) the orientation of the action or movement displayed (forexample, a car going from left to right, a human raising a hand, or adog jumping). In this regard, the overlay model (for example, in theoverlay processor 160) can be trained via the OMTT unit 180.

FIG. 6 shows a nonlimiting example of an operation of the overlayprocessor 160 when provided with the AD content 52 (shown in FIG. 2).Here, the overlay processor 160 can analyze the image data in the ADcontent 52, including any existing special effects, and identify theobject in the image 32 as a building structure. The overlay processor160 can identify (or predict) each part of the object in the image 32,including a position and a directional orientation. The overlayprocessor 160 can identify (or predict) the edges and/or outlines of theobject. The overlay processor 160 can be arranged to identify (orpredict) any changes relating to the object as a function of time,including for example, motion, direction of motion, speed of motion.Simultaneously, the overlay processor 160 can identify (or predict) thebackground in the image data. Based on an analysis of the object andbackground, the overlay processor 160 can determine (or predict) anaction in the AD content, such as, for example, a thing or a directionthe object is point toward.

The object can include, for example, an animal, an article, a product,or any item that is to be emphasized, brought into the foreground orhighlighted during rendering or display on the end-user communicationdevice 40. The background can include any thing or things, other thanthe object, that can be deemphasized or played down during rendering ordisplay on the end-user communication device 40. For instance, in theexample shown in FIG. 6, the object is a building and the background iseverything else surrounding the building.

Based on the identification (or prediction) of the object and/orbackground, the overlay processor 160 can create or modify a specialeffect to enhance the AD content to emphasize, highlight or otherwisedraw attention to the image 34.

FIG. 7 shows a nonlimiting example of a modification of the AD contentperformed by the overlay processor 160. As seen, the special effect 57,which in this nonlimiting example is an overlay border, can be modifiedsuch that it does not overlay any part of the object in the image 32. Inthis example, the overlay processor 160 has also resized and/orrepositioned the object such that the special effect 57 remains visibleto emphasize the object, textual message 34 and call-to-action selectors56.

Referring to FIG. 5, the overlay processor 160 can be arranged toanalyze the image data in the AD content and determine (or predict)display variables of an object (or background) in the image data, suchas, for example, size, position, outline, directional orientation, orany other attribute of the object that can be perceived or viewed whenrendered on the display device 354 (shown in FIG. 4) of the end-usercommunication device 40. Based on the analysis of the image data andidentification (or prediction) of the object and background, the overlayprocessor 160 can determine display properties for the special effect 57to be rendered with the AD content on the display device 354 (shown inFIG. 4), such that the object (or background) is emphasized, brought tothe foreground or highlighted. The display properties can include, forexample, shape (for example, rectangle, square, oval, circle, triangle,box, sphere, pyramid, etc.), size (for example, height, width, depth,radius, etc.), color, contrast, brightness, shading, texture, orlocation, or image manipulation, such as, for example, zoom-in/out,blur/focus, or shake/vibrate of the image. The overlay model can bebuilt, trained or tuned by the overlay model trainer 180.

The OMTT unit 180 can train the overlay model using annotated historicaldata, including the training dataset stored in the DB 120E, and tune theoverlay model using the testing dataset stored in the DB 120E. The OMTTunit 180 can tune the overlay model during operation of the MCM system100 by updating parametric values in the overlay model to, for example,minimize any differences between the overlay model predictions andactual, observed results.

In a nonlimiting embodiment the OMTT unit 180 can be arranged tointeract with an operator (for example, via the IO interface 140, shownin FIG. 5) to create training datasets, or the OMTT unit 180 can beprovided with a large training dataset of AD content, such as, forexample, sets of tens, hundreds, thousands, hundreds of thousands, ormore unique AD content records. Each AD content record can include imageframe pixels that can be divided into a plurality image blocks, whicheach block corresponding to a unique region of the image frame, withoutany overlapping pixels. The image block can include, for example, atwo-dimensional a×b array of pixels, where a is a number of pixelslocated consecutively along a row of image pixels and b is a number ofpixels located consecutively along a column of image pixels, where a andb are positive integers greater than 1, and where a and b can have thesame or different values. Alternatively, the image pixels in the imageframe can be divided such that the image blocks have differentdimensions from each other. The image block can be scaled such that itcannot comprise more than one object per image block. Each image blockcan include a unique address with respect to the image frame.

The OMTT unit 180 (shown in FIG. 5) can be arranged to receive object-,background- or special effect-specific parameters for each aberration,background, and special effect when creating a training dataset. The MCMsystem 100 can be arranged to allow a user to annotate AD content (forexample, using a mouse or stylus) to mark a plurality of points on thedisplay screen, which can then be used by the MCM system 100, forexample, through interaction with the processor 110 (shown in FIG. 5),to calculate and determine object type, background type and specialeffect type, including shape, size dimensions, location, orientation,and display properties, such as, for example, color, texture, contrast,brightness, shading, etc. for each portion of the object, background andoverlay.

The annotations made by the user for each AD content record can becommunicated to the OMTT unit 180 (shown in FIG. 5), which can generatemetadata for each AD content record, including the object, backgroundand special effect for the image data. The annotations can becommunicated to the OMTT unit 180 as label tuning commands, or stored inthe storage 120 and associated with corresponding image blocks, whichcan also be stored in the storage 120. The metadata can be embedded inthe image block data and stored as labeled image block data in thestorage 120. The generated metadata can include indexing data for eachobject, background and special effect.

The training dataset, which includes an accumulation of labeled ADcontent records, can be used to create a training database in the DB120D (shown in FIG. 5) or to augment an existing training dataset totrain the overlay model in the overlay processor 160 (shown in FIG. 5)for improved performance.

All the image blocks in an AD content record can be rendered, forexample, by the multimedia content renderer 170 (shown in FIG. 5) or theprocessor 352 (shown in FIG. 4), on a display device (for example,display device 354, shown in FIG. 4) to display the AD content. Themultimedia content renderer 170 can include a computing device or, aspreviously noted, a computer resource that can be executed by theprocessor 110 (shown in FIG. 5) or the processor 352 (shown in FIG. 4).

The multimedia content renderer 170 can be arranged to create,configure, or update the MCR application. The MCR application can becreated, configured or updated in the multimedia communication system 10(shown in FIG. 1) and sent or downloaded to the end-user communicationdevice 40. The multimedia content renderer 170 can include a scriptingmodule that can provide a special run-time environment that can automatethe execution of tasks in the MCM system 100. The scripting module caninclude one or more scripting languages, which can range fromdomain-specific languages to general-purpose languages, such as, forexample, Bash (e.g., for Unix operating systems), ECMAScript (orJavaScript) (e.g., for web browsers), Visual Basic (e.g., for Microsoftapplications), Lua, Python, and the like. The multimedia contentrenderer 170 can include one or more of the scripting modules found at,for example, <<https://developers.google.com/apps-script/>>.

The multimedia content renderer 170 can be configured to cause ADcontent or a webpage to be provided to the end-user communication device40 that includes AD content (for example, AD content 52, shown in FIG. 2or 7) and instructions (e.g., webpage code) that invoke a renderingscript responsive to a request for the AD content or webpage on theend-user communication device 40. The rendering script can be downloadedfrom the multimedia content render 170 to the end-user communicationdevice 40 and stored locally at the device, or it can be provided inreal-time to the end-user communication device 40 with the webpage. Therendering script can be provided as part of the MCR application.

The overlay processor 160 and multimedia content renderer 170 can bearranged to interact with, for example, the processor 110, storage 120or network interface 140, to receive and analyze multimedia content,including AD content. The multimedia content renderer 170 can bearranged to interact with the processor 110 or network interface 140 tosend/receive multimedia content and/or the MCR application.

The MCR application can include computer executable code or instructionsor sets of instructions that, when executed by the processor in theend-user communication device 40, causes the device 40 to render thedisplay content, including a special effect customized to the ADcontent, for example, such that an object in the display content isemphasized or pops-out. The MCR application can include one or morerules for the display of the special effect on the end-usercommunication device 40, including, for example, one or more rules thatdefine how the object and background are to be rendered by the end-usercommunication device 40.

For instance, the rules in the MCR application can define, whendisplaying the special effect, one or more colors, one or more textures,one or more sizes, or other effects that emphasize an object in theimage data. The rules can further define, when rendering the specialeffect, one or more colors, one or more fonts, one or more textures, oneor more sizes, or one or more layouts.

The MCR application can further include rules that define how an objector background is to be displayed on the end-user communication device40, including, for example, dimensions, location and orientation.

The MCR application can include one or more rules that define the layoutfor rendering the AD content, with or without the main display content,on the end-user communication device 40.

In the case of HTML5, the MCR application can include an audio tag(e.g., an instruction to embed an audio file/link in the displayedscreen and how to play it), a video tag (e.g., an instruction to embedvideo in the displayed screen and how to play it), a source tag (e.g.,can be used with audio or video to identify a source for the audio orvideo), an embed tag (e.g., an instruction to embed specified media typefor content that might lack support within other media elements), acanvas tag (e.g., an instruction to set aside part of the displayscreen), a svg tag (e.g., an instruction to embed vector graphics (e.g.,object, text, overlay and/or background) encoded with SVG markuplanguage, to allow graphics (e.g., objects, text, overlay and/orbackground) to be scaled dynamically to the area and shape of thedisplay screen without losing any graphic quality), and the like. Asunderstood by those skilled in the art, the MCR application can includeother tags that, when referenced by, for example, a style sheetlanguage, cause the end-user communication device 40 to render the ADcontent with or without the MD content, to have a location, layout,size, shape, color, texture, font, special effect, backdrop, and thelike, that is optimal for the particular AD content and/or end-usercommunication device 40.

For instance, using the canvas tag in HTML5, a portion of a displayscreen on an end-user communication device 40 can be targeted for acontent slot (or new canvas window) for insertion of the AD content 52(shown in FIG. 7), including the height and width of the canvas whichmight be measured from an upper-left corner of the display screen. Thecanvas window should match the size of the AD content, including overlay57.

FIG. 8 shows a nonlimiting embodiment of a process 200 for analyzing ADcontent and selectively modifying display properties for the AD contentbased on image data included in the AD content. The process 200 can beperformed by the MCM system 100 (shown in FIG. 5). The process 200 canbe performed by the overlay machine learning model, which can beincluded in the MCM system 100 or downloaded to and installed on, forexample, the end-user communication device 40 (shown in FIG. 1). Aspreviously noted, the MCM system 100 (shown in FIG. 5) can be includedin either or both the multimedia communication system 10 and theend-user communication device 40.

Referring to FIGS. 5 and 8 concurrently, auxiliary display (AD) contentcan be received by the MCM system 100 (Step 205). The AD content can bereceived by itself or as part of multimedia content and parsed from themultimedia content. The AD content can be forwarded and applied to aninput of the overlay processor 160 (shown in FIG. 5), wherein image datain the AD content can be analyzed by the overlay processor 160 (Step210). The image data can be analyzed to detect and identify (or predict)any discernible objects or background (Step 215). The image data canalso be analyzed to detect whether any special effect is included in theAD content, as well as any detectable action by the object or backgroundin the AD content (Step 215). Based on the identified (or predicted)objects or background, and identified (or predicted) overlay or actionby the object or background, the overlay processor 160 can determine atype of special effect to apply to display properties for the AD content(Step 220), including NO CUSTOMIZATION (or NON-CUSTOMIZATION), SIMPLEFOREGROUND, or SELECTIVE.

FIGS. 16A to 16C depict a nonlimiting example of an image data analysisthat can be performed by the overlay processor 160 in Step 215. As seen,AD content 52 can be received that includes a main object 32 (forexample, person) and background objects 38 (for example, laptop) and 39(for example, back of second person). The overlay processor 160 cananalyze the image data and, using a combination of depth estimation (forexample, depicted in FIG. 16B) and instance segmentation (for example,depicted in FIG. 16C), obtain an object- and depth-aware layerrepresentation that identifies the objections 32, 38, 39 and theirdistance with respect to the image pickup device (not shown) that wasused to take the photo, or the virtual image pickup device in the caseof machine generated AD content 52. In this example, objects 32, 38 and39 can have inverse depths 0.35, 0.56 and 0.92, respectively. Both depthestimation and instance segmentation can be powered by, for example, aDCNN in the overlay processor 160.

The overlay processor 160 can also use depth to eliminate objects in thebackground, and to refine layer boundaries. For instance, in the exampledepicted in FIGS. 16A to 16C, instance segmentation detected two people(32, 39) and the laptop (38) between them. The overlay processor 160 canuse depth to eliminate the second person 39, who appears to be in thebackground, and also to make the layer boundaries snap more tightly tothe person 32. The special effect 57 can include a border that bindsclosely to the person 32, such as, for example, surrounding the person32, but display a break in the border so that the person's head extendsoutside the frame created by the border, for example, as seen in theexample in FIG. 14.

FIG. 9 depicts another nonlimiting example of AD content 52 analyzed bythe overlay processor 160 and rendered with the special effect 57, whichin this example includes treating a border as a physical object, like aplanar picture frame. The frame can be rendered such that it is placedinto the scene. The frame can be rendered in multiple ways, and havingthe segmentation and the depth layers allows the overlay processor 160to figure out how the occlusions should work out. In this example, theoverlay processor 160 can either place the frame behind the car, orsomewhere in between allowing only a portion of the car to pop out ofthe frame, making it look like the car driving into or out of the frame.

As noted above, the overlay processor 160 can generate and implementother special effects, such as, for example, animating text to make itmove behind a subject, rotating the border,brightening/darkening/blurring the background, or applying differentstylistic effects on the foreground vs. background. This can make astatic image appear to come to life.

FIGS. 10-14 depict further nonlimiting examples of AD content that canbe identified (or predicted) by the overlay processor 160 to includeimage data for which display properties can be modified according to oneof the three exemplary overlay types (Step 220, NO CUSTOMIZATION, SIMPLEFOREGROUND, SELECTIVE FOREGROUND).

For instance, the overlay processor 160 can determine the NOCUSTOMIZATION special effect type (Step 220) if it determines (in Step215) the AD content: does not include any clear or discernibleforeground (for example, shown in FIG. 2) (Case #1); or containsprimarily foreground (for example, shown in FIGS. 11, 13, 14) (Case #2);or contains an action scene (for example, shown in FIG. 12) (Case #3).Cases #1 to #3 are provided for illustrative purposes only, and do notin any way represent the broad range of possible AD content for whichthe NO CUSTOMIZATION special effect type might be selected. Based on thedetected (or predicted) foreground (or object) (Step 220, NOCUSTOMIZATION), the overlay processor 160 can modify the displayproperties for the AD content to either apply a special effect (forexample, overlay border) or edit an existing special effect (forexample, overlay border) in the AD content (Step 225). The modified ADcontent can be rendered (with or without MD content) on the displaydevice (for example, display device 354, shown in FIG. 4), for example,by or in response to data and instructions provided by the multimediacontent renderer 170 (shown in FIG. 5) (Step 250). The modified ADcontent can include a special effect that, when rendered by the end-usercommunication device 40, is displayed, for example, as a border overlayalong the inner edges of the entirety of the content slot on the displayscreen allocated to the AD content (or “AD content display window”), asseen in, for example, FIG. 2.

In Case #1, the AD content might include, for example, only a backdrop(for example, a blue screen), an image of a forest, or any other imagethat lacks any object of interest or focal point. In that instance, theoverlay processor 160 can modify the display properties (in Step 225) tocreate and apply a special effect, such as, for example, an overlayborder, to the AD content without any customization, avoiding overlay ontop of any logo, text, call-to-action selectors or other displayablefeatures that are to be visible to the user (for example, shown in FIG.15). If, however, the received AD content includes an existing specialeffect, such as, for example, an overlay border, then the displayproperties can be modified to ensure the border does not overlay on topof any logo, text, call-to-action, or other displayable features thatshould be visible to the user.

In Case #2, the AD content might include a foreground image (or object)that occupies, for example, more than ninety-percent (90%) of the ADcontent display window, as seen in the example shown in FIG. 11 or 13.In that instance, the overlay processor 160 can modify the displayproperties (in Step 225) to create and apply a partial special effect,such as, for example, a partial overlay border (including an overlayportion and a non-overlay portion) to the AD content such that theborder has at least one corner and the visible part of the border is atleast fifteen-percent (15%) of the total height of the border, as seenin the example, shown in FIG. 11. If, however, the AD content includesan existing special effect (for example, overlay border), then thedisplay properties for the special effect can be modified to provide, inthis nonlimiting example, an overlay portion and a non-overlay portionby, for example, removing one or more portions of the border, such thatthe border has at least one corner and the visible part of the border isat least 15% of the total height of the border. In this particularexample, the overlay processor 160 can determine that there are twopeople in the scene, one person (39) close to the image pickup deviceand one person (32) further away, with the laptop (38) between them.With this information, the overlay processor 160 can guide selectionand/or placement of the special effect, such as, for example, theoverlay border.

In Case #3, the AD content might include an action scene, such as, forexample, shown in FIG. 12. In that instance, where the special effectcomprises an overlay border that might interfere with the actiondepicted in the AD content, the overlay processor 160 can modify thedisplay properties (in Step 225) to enhance the action shown in theimage by, for example, creating and applying a partial overlay border tothe AD content (or removing a portion of preexisting border) so that anoverlay portion and a non-overlay portion emphasize or highlight anaction depicted in the AD content. FIG. 12 shows a nonlimiting exampleof an overlay border that can be applied (or modified) on the image inthe AD content, including a partial overlay portion and a non-overlayportion.

The overlay processor 160 can determine the SIMPLE FOREGROUND specialeffect type (Step 220) if it determines (in Step 215) the AD contentincludes a clear foreground (or object) with a good ratio of foregroundto background in the AD content (for example, shown in FIG. 11) (Case#4). In that instance, the overlay processor 160 can modify the displayproperties for the AD content to apply a partial special effect, suchas, for example, a partial overlay border (Step 230). For example, theoverlay processor 160 can create and apply a partial overlay border, oredit and remove a portion of a preexisting border such that the overlayborder includes a non-overlay portion (or break in the overlay) suchthat the overlay does not overlap the foreground (or object) or aportion thereof, thereby creating an illusion of depth when the ADcontent is rendered on the display device 354 (shown in FIG. 4) of theend-user communication device 40. The modified AD content can berendered (with or without MD content) on the display device 354 (shownin FIG. 4), for example, by, or in response to data and instructionsprovided by the multimedia content renderer 170 (shown in FIG. 5) (Step250).

The overlay processor 160 can determine the SELECTIVE FOREGROUND overlaytype (Step 220) if it determines (in Step 215) the AD content includes:an object, such as, for example, a product (e.g., vehicle) that has adirectional orientation (for example, shown in FIG. 9) (Case #5); or anobject, such as, for example, an animal (e.g., a portrait of a human, acat, a dog, etc.) (for example, shown in FIG. 11) (Case #6); or anaction scene wherein the action can be enhanced, for example, by zoomingin or out a portion of the action scene (for example, shown in FIG. 12)(Case #7).

In Case #5, the AD content might include, for example, a vehicle (forexample, shown in FIG. 9), in which case the overlay processor 160 candetermine the directional orientation of the vehicle (in Step 215) andmodify the display properties for the AD content to create and apply, orto edit an existing special effect, such as, for example, an overlayborder, in the AD content (Step 235). Based on the object and itsdirectional orientation, the overlay processor 160 can modify thedisplay properties of the AD content to apply a partial overlay border(or remove a portion of an existing border) to provide an overlayportion and a non-overlay portion that create an illusion of motion ordynamics to the object, for example, as shown in FIG. 9, 12 or 14.

In Case #6, the AD content might include, for example, a portrait of aperson (for example, shown in FIG. 11 or 13), in which case the overlayprocessor 160 can determine an outline and/or directional orientation ofthe person (in Step 215) and modify the display properties for the ADcontent to create and apply, or to edit an existing special effect, suchas, for example, an overlay border, in the AD content (Step 235). Basedon the position of the person and/or directional orientation, theoverlay processor 160 can modify the display properties of the ADcontent to apply a partial overlay border, or remove a portion of anexisting border that overlay the person's head, as shown in FIGS. 11 and13.

In Case #7, the AD content might include, for example, an action scenewherein the action can be enhanced by a customizable special effect (forexample, an overlay border as seen in FIG. 12) or complemented withanother special effect, such as, for example, zooming in or out aportion of the action scene (for example, shown in FIG. 13). In theformer instance, the overlay processor 160 can modify the displayproperties (in Step 235) to enhance the action shown in the image by,for example, creating and applying a partial overlay border to the ADcontent (or removing a portion of preexisting border) so that action isemphasized, as shown, for example, in FIG. 12.

In the latter instance, a determination can be made by the overlayprocessor 160 whether a portion of the image in the AD content should bemodified to further emphasize the action the AD content (Step 240). If,for example, the overlay processor 160 determines that an action (forexample, shown in FIG. 12 or 13) can be further emphasized by applying aspecial effect such as, for example, zooming in or out a portion of theimage (YES at Step 240), then the display properties for the AD contentcan be further modified to modify the image (Step 245), in addition tomodification or application of the overlay border special effect (Step235); otherwise (NO at Step 245), the modified AD content can berendered (with or without MD content) on the display device 354 (shownin FIG. 4), for example, by, or in response to data and instructionsprovided by the multimedia content renderer 170 (shown in FIG. 5) (Step250).

The terms “a,” “an,” and “the,” as used in this disclosure, means “oneor more,” unless expressly specified otherwise.

The term “backbone,” as used in this disclosure, means a transmissionmedium that interconnects one or more computing devices or communicatingdevices to provide a path that conveys data signals and instructionsignals between the one or more computing devices or communicatingdevices. The backbone can include a bus or a network. The backbone caninclude an ethernet TCP/IP. The backbone can include a distributedbackbone, a collapsed backbone, a parallel backbone or a serialbackbone.

The term “bus,” as used in this disclosure, means any of several typesof bus structures that can further interconnect to a memory bus (with orwithout a memory controller), a peripheral bus, or a local bus using anyof a variety of commercially available bus architectures. The term “bus”can include a backbone.

The term “communication device,” as used in this disclosure, means anyhardware, firmware, or software that can transmit or receive datapackets, instruction signals, data signals or radio frequency signalsover a communication link. The communicating device can include acomputer or a server. The communicating device can be portable orstationary.

The term “communication link,” as used in this disclosure, means a wiredor wireless medium that conveys data or information between at least twopoints. The wired or wireless medium can include, for example, ametallic conductor link, a radio frequency (RF) communication link, anInfrared (IR) communication link, or an optical communication link. TheRF communication link can include, for example, WiFi, WiMAX, IEEE302.11, DECT, OG, 1G, 2G, 3G, 4G, or 5G cellular standards, orBluetooth. A communication link can include, for example, an RS-232,RS-422, RS-485, or any other suitable serial interface.

The terms “computer,” “computing device,” or “processor,” as used inthis disclosure, means any machine, device, circuit, component, ormodule, or any system of machines, devices, circuits, components, ormodules that are capable of manipulating data according to one or moreinstructions. The terms “computer,” “computing device” or “processor”can include, for example, without limitation, a processor, amicroprocessor (μC), a central processing unit (CPU), a graphicprocessing unit (GPU), an application specific integrated circuit(ASIC), a general purpose computer, a super computer, a personalcomputer, a laptop computer, a palmtop computer, a notebook computer, adesktop computer, a workstation computer, a server, a server farm, acomputer cloud, or an array or system of processors, μCs, CPUs, GPUs,ASICs, general purpose computers, super computers, personal computers,laptop computers, palmtop computers, notebook computers, desktopcomputers, workstation computers, or servers.

The terms “computing resource” or “computer resource,” as used in thisdisclosure, means software, a software application, a web application, aweb page, a computer application, a computer program, computer code,machine executable instructions, firmware, or a process that can bearranged to execute on a computing device as one or more computingresource processes.

The term “computing resource process,” as used in this disclosure, meansa computing resource that is in execution or in a state of beingexecuted on an operating system of a computing device. Every computingresource that is created, opened or executed on or by the operatingsystem can create a corresponding “computing resource process.” A“computing resource process” can include one or more threads, as will beunderstood by those skilled in the art.

The terms “computer resource asset” or “computing resource asset,” asused in this disclosure, means a computing resource, a computing deviceor a communicating device, or any combination thereof.

The term “computer-readable medium,” as used in this disclosure, meansany non-transitory storage medium that participates in providing data(for example, instructions) that can be read by a computer. Such amedium can take many forms, including non-volatile media and volatilemedia. Non-volatile media can include, for example, optical or magneticdisks and other persistent memory. Volatile media can include dynamicrandom-access memory (DRAM). Common forms of computer-readable mediainclude, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, any other magnetic medium, a CD-ROM, DVD, any otheroptical medium, punch cards, paper tape, any other physical medium withpatterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any othermemory chip or cartridge, a carrier wave as described hereinafter, orany other medium from which a computer can read. The computer-readablemedium can include a “cloud,” which can include a distribution of filesacross multiple (e.g., thousands of) memory caches on multiple (e.g.,thousands of) computers.

Various forms of computer readable media can be involved in carryingsequences of instructions to a computer. For example, sequences ofinstruction (i) can be delivered from a RAM to a processor, (ii) can becarried over a wireless transmission medium, or (iii) can be formattedaccording to numerous formats, standards or protocols, including, forexample, WiFi, WiMAX, IEEE 302.11, DECT, OG, 1G, 2G, 3G, 4G, or 5Gcellular standards, or Bluetooth.

The term “database,” as used in this disclosure, means any combinationof software or hardware, including at least one computing resource or atleast one computer. The database can include a structured collection ofrecords or data organized according to a database model, such as, forexample, but not limited to at least one of a relational model, ahierarchical model, or a network model. The database can include adatabase management system application (DBMS). The at least oneapplication may include, but is not limited to, a computing resourcesuch as, for example, an application program that can accept connectionsto service requests from communicating devices by sending back responsesto the devices. The database can be configured to run the at least onecomputing resource, often under heavy workloads, unattended, forextended periods of time with minimal or no human direction.

The terms “including,” “comprising” and their variations, as used inthis disclosure, mean “including, but not limited to,” unless expresslyspecified otherwise.

The term “network,” as used in this disclosure means, but is not limitedto, for example, at least one of a personal area network (PAN), a localarea network (LAN), a wireless local area network (WLAN), a campus areanetwork (CAN), a metropolitan area network (MAN), a wide area network(WAN), a metropolitan area network (MAN), a wide area network (WAN), aglobal area network (GAN), a broadband area network (BAN), a cellularnetwork, a storage-area network (SAN), a system-area network, a passiveoptical local area network (POLAN), an enterprise private network (EPN),a virtual private network (VPN), the Internet, or the like, or anycombination of the foregoing, any of which can be configured tocommunicate data via a wireless and/or a wired communication medium.These networks can run a variety of protocols, including, but notlimited to, for example, Ethernet, IP, IPX, TCP, UDP, SPX, IP, IRC,HTTP, FTP, Telnet, SMTP, DNS, ARP, ICMP.

The term “server,” as used in this disclosure, means any combination ofsoftware or hardware, including at least one computing resource or atleast one computer to perform services for connected communicatingdevices as part of a client-server architecture. The at least one serverapplication can include, but is not limited to, a computing resourcesuch as, for example, an application program that can accept connectionsto service requests from communicating devices by sending back responsesto the devices. The server can be configured to run the at least onecomputing resource, often under heavy workloads, unattended, forextended periods of time with minimal or no human direction. The servercan include a plurality of computers configured, with the at least onecomputing resource being divided among the computers depending upon theworkload. For example, under light loading, the at least one computingresource can run on a single computer. However, under heavy loading,multiple computers can be required to run the at least one computingresource. The server, or any if its computers, can also be used as aworkstation.

The term “transmission” or “transmit,” as used in this disclosure, meansthe conveyance of data, data packets, computer instructions, or anyother digital or analog information via electricity, acoustic waves,light waves or other electromagnetic emissions, such as those generatedwith communications in the radio frequency (RF) or infrared (IR)spectra. Transmission media for such transmissions can include coaxialcables, copper wire and fiber optics, including the wires that comprisea system bus coupled to the processor.

Devices that are in communication with each other need not be incontinuous communication with each other unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

Although process steps, method steps, or algorithms may be described ina sequential or a parallel order, such processes, methods and algorithmsmay be configured to work in alternate orders. In other words, anysequence or order of steps that may be described in a sequential orderdoes not necessarily indicate a requirement that the steps be performedin that order; some steps may be performed simultaneously. Similarly, ifa sequence or order of steps is described in a parallel (orsimultaneous) order, such steps can be performed in a sequential order.The steps of the processes, methods or algorithms described in thisspecification may be performed in any order practical.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.The functionality or the features of a device may be alternativelyembodied by one or more other devices which are not explicitly describedas having such functionality or features.

What is claimed is:
 1. A multimedia communication system fortransmitting auxiliary display content to an end-user communicationdevice to be rendered on a display device with a special effect toemphasize an image included in the auxiliary display content, themultimedia communication system comprising: a processor that is arrangedto: analyze image data included in an auxiliary display content; detectat least a foreground image and a background image in the auxiliarydisplay content based on the analysis of the image data; determine aspecial effect to emphasize the foreground image based on the analysisof the image data, wherein the special effect comprises a predeterminedshape with a plurality of portions overlapping with the foreground imageand at least one portion overlapping with a portion of the backgroundimage; and apply the special effect to the auxiliary display content tomodify display properties for the auxiliary display content by addingthe predetermined shape such that at least one of the plurality ofportions overlapping with the foreground image is a non-overlay portionand at least another one of the plurality of portions overlapping withthe foreground image is an overlay portion, wherein the foreground imageextends from at least an outer side of the overlay portion and the atleast one portion of the predetermined shape overlapping with theportion of the background image overlays the portion of the backgroundimage; and, a transmitter arranged to send the auxiliary display contentwith modified display properties to an end-user communication device. 2.The multimedia communication system of claim 1, wherein the processor isarranged to: generate or modify a multimedia content renderingapplication; and send the multimedia content rendering application tothe end-user communication device, wherein the multimedia contentrendering application is arranged to, when executed by a processor inthe end-user communication device, render the auxiliary display contentwith modified display properties on a display device.
 3. The multimediacommunication system of claim 1, wherein the special effect comprises anon-customization special effect that comprises an overlay border that,when rendered by the end-user communication device, is displayed aroundan inner-border of an auxiliary display content display window on adisplay device.
 4. The multimedia communication system of claim 3,wherein the overlay border has at least one corner and the non-overlayportion that emphasize an object image in the auxiliary display content.5. The multimedia communication system of claim 1, wherein the specialeffect comprises a simple foreground special effect that comprises anoverlay border having an overlay portion and the non-overlay portionthat, when rendered by the end-user communication device, is displayedas an overlay border atop of the background image in an auxiliarydisplay content display window, with an object image being displayedwithout any overlay in the auxiliary display content display window. 6.The multimedia communication system of claim 5, wherein the overlayborder has at least one corner and the non-overlay portion is renderedto emphasize the object image or an action depicted in the auxiliarydisplay content.
 7. The multimedia communication system of claim 1,wherein the special effect comprises a selective foreground specialeffect that comprises an overlay border having an overlay portion andthe non-overlay portion that, when rendered by the end-usercommunication device, is displayed as an overlay border atop of thebackground image in an auxiliary display content display window, with atleast a portion of an object image being displayed without any overlayin the auxiliary display content display window.
 8. The multimediacommunication system of claim 1, wherein the special effect comprises aselective foreground special effect that comprises a zoom-in or azoom-out of an object image, or a blurring of the background image whenrendered by the end-user communication device.
 9. The multimediacommunication system of claim 8, wherein the selective foregroundspecial effect further comprises zooming-in or zooming-out an objectimage when rendered by the end-user communication device.
 10. Acomputer-implemented method for transmitting auxiliary display contentto an end-user communication device to be rendered on a display devicewith a special effect to emphasize an image included in the auxiliarydisplay content, the method comprising: analyzing, by a processor, imagedata included in an auxiliary display content; detecting, by theprocessor, at least a foreground image and a background image in theauxiliary display content; determining, by the processor, a specialeffect to emphasize the foreground image based on the analysis of theimage data, wherein the special effect comprises a predetermined shapewith a plurality of portions overlapping with the foreground image andat least one portion overlapping with a portion of the background image;and applying, by the processor, the special effect to the auxiliarydisplay content to modify display properties for the auxiliary displaycontent by adding the predetermined shape such that at least one of theplurality of portions overlapping with the foreground image is anon-overlay portion and at least another one of the plurality ofportions overlapping with the foreground image is an overlay portion,wherein the foreground image extends from at least an outer side of theoverlay portion and the at least one portion of the predetermined shapeoverlapping with the portion of the background image overlays theportion of the background image.
 11. The method of claim 10, furthercomprising: creating a multimedia content rendering application; andsending the multimedia content rendering application to an end-usercommunication device, wherein the multimedia content renderingapplication is arranged to, when executed by a processor in the end-usercommunication device, render the auxiliary display content with modifieddisplay properties on a display device.
 12. The method of claim 10,wherein the special effect comprises a non-customization special effectthat comprises an overlay border that, when rendered by an end-usercommunication device, is displayed around an inner-border of anauxiliary display content display window on a display device.
 13. Themethod of claim 12, wherein the overlay border has at least one cornerand the non-overlay portion that emphasize an object image in theauxiliary display content.
 14. The method of claim 10, wherein thespecial effect comprises a simple foreground special effect thatcomprises an overlay border having an overlay portion and thenon-overlay portion that, when rendered by an end-user communicationdevice, is displayed as an overlay border atop of the background imagein an auxiliary display content display window, with an object imagebeing displayed without any overlay in the auxiliary display contentdisplay window.
 15. The method of claim 14, wherein the overlay borderhas at least one corner and the non-overlay portion is rendered toemphasize the object image or an action depicted in the auxiliarydisplay content.
 16. The method of claim 10, wherein the special effectcomprises a selective foreground special effect that comprises anoverlay border having an overlay portion and the non-overlay portionthat, when rendered by an end-user communication device, is displayed asan overlay border atop of the background image in an auxiliary displaycontent display window, with at least a portion of an object image beingdisplayed without any overlay in the auxiliary display content displaywindow.
 17. The method of claim 10, wherein the special effect comprisesa selective foreground special effect that comprises a zoom-in or azoom-out of an object image when rendered by an end-user communicationdevice, or a blurring of the background image when rendered by theend-user communication device.
 18. The method of claim 17, wherein theselective foreground special effect further comprises zooming-in orzooming-out an object image when rendered by the end-user communicationdevice.